{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 如何使用千帆 Python SDK 对模型进行评估\n",
    "\n",
    "在 0.3.0 版本中，千帆 Python SDK 增加了模型评估的功能，支持用户通过 SDK 发起平台在线评估任务，或者本地 API 评估，并且在本地获取到评估报告。\n",
    "\n",
    "# 准备工作\n",
    "\n",
    "在开始之前，请确保你的千帆 Python SDK 已经升级到了 0.3.0 及以上版本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "#-# cell_skip\n",
    "! pip install -U \"qianfan>=0.3.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "并且在环境变量中设置好 Access Key 与 Secret Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-26T03:45:40.178308Z",
     "start_time": "2024-02-26T03:45:40.168742Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# os.environ['QIANFAN_ACCESS_KEY'] = 'your_access_key'\n",
    "# os.environ['QIANFAN_SECRET_KEY'] = 'your_secret_key'\n",
    "\n",
    "# 使用 Service 对象进行评估时，请按实际情况填写 QPS LIMIT，\n",
    "# 取值为所有 Service QPS Limit中的最小值\n",
    "os.environ[\"QIANFAN_QPS_LIMIT\"] = \"1\"\n",
    "os.environ['QIANFAN_LLM_API_RETRY_COUNT'] = \"3\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正文\n",
    "\n",
    "目前 Evaluation 模块支持用户对 `Model` 和 `Service` 对象进行离线或者在线的评估。\n",
    "\n",
    "## 在线评估\n",
    "\n",
    "在线评估依托千帆平台的资源，在平台上运行评估任务。目前在线评估仅支持对 `Model` 对象进行评估，并且要求被评估的数据集已经上传到千帆平台并处于发布状态。\n",
    "\n",
    "下面展示了如何从本地上传一个数据集到千帆并且发布该数据集。首先，我们需要先加载数据集到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'prompt': '地球的自转周期是多久？', 'response': [['大约24小时']]}]\n"
     ]
    }
   ],
   "source": [
    "from typing import Any, Dict\n",
    "\n",
    "from qianfan.dataset import Dataset\n",
    "\n",
    "# 处理数据集用的 map 函数\n",
    "def dataset_map_func(entry: Dict[str, Any]) -> Dict[str, Any]:\n",
    "    entry[\"response\"] = [[entry[\"response\"]]]\n",
    "    return entry\n",
    "\n",
    "# 首先加载数据集\n",
    "ds = Dataset.load(data_file=\"data_file/demo_csv.csv\")\n",
    "# 然后对数据集进行 map 操作，再添加 group 列，并转换成千帆平台的数据集格式\n",
    "ds = ds.map(dataset_map_func).add_default_group_column()\n",
    "ds.pack()\n",
    "\n",
    "print(ds[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们再将该数据集上传到千帆平台并发布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from qianfan.dataset import DataTemplateType, DataStorageType\n",
    "\n",
    "dataset_name = \"random_name\"\n",
    "dataset_template_type = DataTemplateType.NonSortedConversation\n",
    "dataset_storage_type = DataStorageType.PrivateBos\n",
    "bos_storage_id = \"your_bucket_name\"\n",
    "bos_storage_path = \"/test/\"\n",
    "\n",
    "ds = ds.save(qianfan_dataset_create_args={\n",
    "        \"name\": dataset_name,\n",
    "        \"template_type\": dataset_template_type,\n",
    "        \"storage_type\": dataset_storage_type,\n",
    "        \"storage_id\": bos_storage_id,\n",
    "        \"storage_path\": bos_storage_path,\n",
    "    }, \n",
    "    does_release=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据集发布后不可再修改。后续用户如果想复用该数据集进行评估或训练，可以使用 `load()` 方法在本地加载平台数据集的代理对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "cloud_datset_id = ds.inner_data_source_cache.id\n",
    "\n",
    "qianfan_ds = Dataset.load(qianfan_dataset_id=cloud_datset_id)\n",
    "print(len(qianfan_ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在数据集发布成功之后，我们就可以使用该数据集对象进行在线评估了。\n",
    "\n",
    "以 ERNIE-Bot-turbo 预置模型为例，我们首先在 SDK 中创建对应的 `Model` 对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.model import Model\n",
    "\n",
    "eb_turbo_model = Model(version_id=\"amv-qb8ijukaish3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们需要创建我们在线评估所需要使用千帆评估器对象，它们都存在于 `qianfan.evaluation.evaluator` 模块中，一共有三种千帆评估器：\n",
    "\n",
    "+ `QianfanRefereeEvaluator`: 千帆平台的裁判员评估器，使用大模型对被评估大模型的回答打分，可以自定打分 prompt，打分步骤等信息\n",
    "+ `QianfanRuleEvaluator`: 千帆平台的规则评估器，会根据评估结果计算出一系列统计指标\n",
    "+ `QianfanManualEvaluator`: 千帆平台的人工评估器，使用人工打分的方式对被评估大模型的回答打分。使用该评估器的评估需要首先在千帆平台上完成人工打分才能得到最终结果\n",
    "\n",
    "为了演示起见，本例使用 `QianfanRefereeEvaluator` 与 `QianfanRuleEvaluator` 评估器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.evaluation.evaluator import QianfanRefereeEvaluator, QianfanRuleEvaluator\n",
    "from qianfan.evaluation.consts import QianfanRefereeEvaluatorDefaultMetrics, QianfanRefereeEvaluatorDefaultSteps, QianfanRefereeEvaluatorDefaultMaxScore\n",
    "\n",
    "your_app_id = 1\n",
    "\n",
    "qianfan_evaluators = [\n",
    "    QianfanRefereeEvaluator(\n",
    "        app_id=your_app_id,\n",
    "        prompt_metrics=QianfanRefereeEvaluatorDefaultMetrics,\n",
    "        prompt_steps=QianfanRefereeEvaluatorDefaultSteps,\n",
    "        prompt_max_score=QianfanRefereeEvaluatorDefaultMaxScore,\n",
    "    ),\n",
    "    QianfanRuleEvaluator(using_accuracy=True, using_similarity=True),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们需要创建一个 `EvaluationManager` 来启动任务，传入之前创建的 `Model` 对象和 `Dataset` 对象来调用 `eval` 方法\n",
    "\n",
    "`eval` 方法会返回一个 `EvaluationResult` 对象，该对象内部包含了一个 `Dataset` 对象（包括了每条评估条目的输入输出数据，以及单条的评估指标）以及 `metrics` 字典对象（包含了总体的评价指标）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.evaluation import EvaluationManager\n",
    "\n",
    "em = EvaluationManager(qianfan_evaluators=qianfan_evaluators)\n",
    "result = em.eval([eb_turbo_model], ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用户可以按照自己的想法来处理得到的 `EvaluationResult` 对象，如打印："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ERNIE-Bot-turbo': {'accuracy': 0.6666667, 'f1Score': 0.014610389, 'rouge_1': 0.04464286, 'rouge_2': 0, 'rouge_l': 0.01521164, 'bleu4': 0.0026583946, 'avgJudgeScore': 4.3333335, 'stdJudgeScore': 0.94280905, 'medianJudgeScore': 5, 'scoreDistribution': {'-1': 0, '0': 0, '1': 0, '2': 0, '3': 1, '4': 0, '5': 2}, 'manualAvgScore': 0, 'goodCaseProportion': 0, 'subjectiveImpression': '', 'manualScoreDistribution': None}}\n"
     ]
    }
   ],
   "source": [
    "print(result.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者展示得到的 `Dataset` 对象并保存："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_ds = result.result_dataset\n",
    "print(eval_ds.list())\n",
    "local_path = './result/dataset.jsonl'\n",
    "os.makedirs(os.path.dirname(local_path), exist_ok=True)\n",
    "eval_ds.save(data_file=local_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 离线评估\n",
    "\n",
    "离线评估与在线评估不同，它不依赖平台对批量推理得到的数据集进行打分评估，而是把这一过程转移到了本地。借由此，我们可以直接使用本地数据集进行评估，并且可以查看单条数据的评估结果。\n",
    "\n",
    "首先我们需要再次加载数据集，与之前不同的是，离线评估需要手动指定输入和输出列列名"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'prompt': '地球的自转周期是多久？', 'response': '大约24小时'}\n"
     ]
    }
   ],
   "source": [
    "from qianfan.dataset import Dataset\n",
    "\n",
    "ds = Dataset.load(data_file=\"data_file/demo_csv.csv\", input_columns=[\"prompt\"], reference_column=\"response\")\n",
    "\n",
    "print(ds[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们需要创建一个用来评估的对象。与上面的 `Model` 对象不同，这次我们创建一个 `Service` 对象，它代表的是服务的概念。此处我们创建了一个预置服务的 `Service` 对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] [01-05 15:41:19] model.py:331 [t:8120441664]: service type should be specified before exec\n"
     ]
    }
   ],
   "source": [
    "from qianfan.model import Service\n",
    "\n",
    "eb_turbo_service = Service(model=\"ERNIE-Bot-turbo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenCompass 评估器\n",
    "\n",
    "千帆 Python SDK 集成了对 OpenCompass 评估器的支持，如果用户不想手动实现 `LocalEvaluator` ，可以直接使用 opencompass 中的评估器对数据集进行评估。为了使用 OpenCompass 评估器，我们需要先安装 opencompass 包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "#-# cell_skip\n",
    "! pip install opencompass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后在代码中，使用 `OpenCompassLocalEvaluator` 来包裹它们，转换为千帆 Python SDK 的评估器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from opencompass.openicl.icl_evaluator import AccEvaluator\n",
    "\n",
    "from qianfan.evaluation.opencompass_evaluator import OpenCompassLocalEvaluator\n",
    "\n",
    "\n",
    "local_evaluators = [OpenCompassLocalEvaluator(open_compass_evaluator=AccEvaluator())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之后就可以像在线评估一样，使用 `EvaluationManager` 去启动一个本地评估任务"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using the latest cached version of the module from /Users/pengyiyang/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--accuracy/f887c0aab52c2d38e1f8a215681126379eca617f96c447638f751434e8e65b14 (last modified on Thu Jan  4 17:38:16 2024) since it couldn't be found locally at evaluate-metric--accuracy, or remotely on the Hugging Face Hub.\n",
      "Using the latest cached version of the module from /Users/pengyiyang/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--accuracy/f887c0aab52c2d38e1f8a215681126379eca617f96c447638f751434e8e65b14 (last modified on Thu Jan  4 17:38:16 2024) since it couldn't be found locally at evaluate-metric--accuracy, or remotely on the Hugging Face Hub.\n",
      "Using the latest cached version of the module from /Users/pengyiyang/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--accuracy/f887c0aab52c2d38e1f8a215681126379eca617f96c447638f751434e8e65b14 (last modified on Thu Jan  4 17:38:16 2024) since it couldn't be found locally at evaluate-metric--accuracy, or remotely on the Hugging Face Hub.\n"
     ]
    }
   ],
   "source": [
    "from qianfan.evaluation import EvaluationManager\n",
    "\n",
    "em = EvaluationManager(local_evaluators=local_evaluators)\n",
    "result = em.eval([eb_turbo_service], ds, is_chat_service=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后查看得到的评估结果集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'input_prompt': '地球的自转周期是多久？', 'expected_output': '大约24小时', 'model_content': [{'accuracy': 0.0, 'content': '地球的自转周期是**23小时56分**。', 'llm_tag': 'None_None_ERNIE-Bot-turbo'}]}, {'input_prompt': '人类的基本单位是什么？', 'expected_output': '人类', 'model_content': [{'accuracy': 0.0, 'content': '人类的基本单位是**个体**。', 'llm_tag': 'None_None_ERNIE-Bot-turbo'}]}, {'input_prompt': '太阳系中最大的行星是哪颗？', 'expected_output': '木星', 'model_content': [{'accuracy': 0.0, 'content': '太阳系中最大的行星是木星。\\n\\n木星是太阳系中最大的行星，其质量为太阳的千分之一，达到太阳系总质量的十分之一。它也是太阳系中最亮的一颗行星，尤其是在漆黑的夜晚，我们可以轻易地看到它。\\n\\n它不仅是最亮的行星，也是在视觉上最迷人的行星。其大红斑是一个巨大的旋风，直径达到了地球的四倍。此外，木星还是一个天然的实验室，可以帮助科学家更好地了解行星大气层、磁场和重力场的性质。', 'llm_tag': 'None_None_ERNIE-Bot-turbo'}]}]\n"
     ]
    }
   ],
   "source": [
    "result_dataset = result.result_dataset\n",
    "print(result_dataset.list())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上评估结果仅做演示用途。我们还可以以对话的形式对数据集进行评估，这种方式既支持普通的非对话数据集，也支持对话数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'input_prompt': '地球的自转周期是多久？', 'expected_output': '大约24小时', 'model_content': [{'accuracy': 0.0, 'content': '地球的自转周期约为**23小时56分**。', 'llm_tag': 'None_None_ERNIE-Bot-turbo'}]}, {'input_prompt': '人类的基本单位是什么？', 'expected_output': '人类', 'model_content': [{'accuracy': 0.0, 'content': '人类的基本单位是**个体**。', 'llm_tag': 'None_None_ERNIE-Bot-turbo'}]}, {'input_prompt': '太阳系中最大的行星是哪颗？', 'expected_output': '木星', 'model_content': [{'accuracy': 0.0, 'content': '太阳系中最大的行星是木星。\\n\\n木星是太阳系中最大的行星，其质量为太阳的千分之一，是太阳系中其他七大行星质量总和的2.5倍。木星主要由氢和氦组成，中心温度估计高达30，000℃以上，没有陆地，几乎全部为液态氢分子覆盖着。木星非常巨大，赤道直径为139，822公里，约为地球的11倍。它的体积和巨大的物质构成使其成为太阳系中最有可能存在生命或其他形式的物质结构的行星之一。', 'llm_tag': 'None_None_ERNIE-Bot-turbo'}]}]\n"
     ]
    }
   ],
   "source": [
    "from qianfan.evaluation import EvaluationManager\n",
    "\n",
    "em = EvaluationManager(local_evaluators=local_evaluators)\n",
    "result = em.eval([eb_turbo_service], ds)\n",
    "result_dataset = result.result_dataset\n",
    "\n",
    "print(result_dataset.list())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到的数据集还可以保存到本地文件中，以做它用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_dataset.save(data_file=\"data_file/eval_result.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于上文提到的在线评估方式，用户也可以通过将 `QianfanEvaluator` 替换为 `LocalEvaluator` 来将评估流程转移到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'input_prompt': '人类的基本单位是什么？', 'expected_output': '人类', 'model_content': [{'accuracy': 0.0, 'content': '人类社会学界、群落、组织、团队、组织、机构、公司、政府、公司、家庭、家庭、军队、学校和家庭、村落、门派、门派、门派、门派、门派是人类的集合。\\n\\n\\n\\n\\n人类的基本结构层次单位是集合名词，人类的基本单位是集合名词，人类的基本单位是集合名词，人类的基本单位是集合名词，人类的基本单位是个人。\\n\\n\\n人类的基本单位是基本单位是个人。', 'llm_tag': '2_7170_None'}]}, {'input_prompt': '地球的自转周期是多久？', 'expected_output': '大约24小时', 'model_content': [{'accuracy': 0.0, 'content': '地球自转一圈的时间是一天。', 'llm_tag': '2_7170_None'}]}, {'input_prompt': '太阳系中最大的行星是哪颗？', 'expected_output': '木星', 'model_content': [{'accuracy': 0.0, 'content': '木木木木木木木木木木木木木木木木木木木木木木木木木木木木木木木木木星。\\n\\n\\n\\n太阳系中最大的行星是木星是木星，其直径约为地球直径的11111. 4022倍，5倍，比地球的1218倍。', 'llm_tag': '2_7170_None'}]}]\n"
     ]
    }
   ],
   "source": [
    "from qianfan.evaluation import EvaluationManager\n",
    "\n",
    "cloud_datset_id = ds.inner_data_source_cache.id\n",
    "qianfan_ds = Dataset.load(qianfan_dataset_id=cloud_datset_id)\n",
    "\n",
    "em = EvaluationManager(local_evaluators=local_evaluators)\n",
    "result = em.eval([eb_turbo_model], qianfan_ds)\n",
    "\n",
    "result_dataset = result.result_dataset\n",
    "print(result_dataset.list())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bce-qianfan-sdk",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
